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INTRODUCTION 

In question and answer format, this digest illustrates the variety of basic and theoretical 
issues in evaluation with which aspiring evaluators should be conversant in order to 
claim they know the knowledge base of their profession. Please note that none of these 
questions have a single correct answer and space limitations prevent providing the level 
of detailed discussion that each deserves. The questions vary considerably in difficulty 
and in how universally the issues involved would be recognized by most evaluators 
today. What follows, therefore, are outlines of the issues rather than correct "answers." 
For more extensive information on the topics discussed in these questions, please refer 
to the references found at the end of this digest, especially Shadish, Cook, and Leviton 
(1991). 

WHAT ARE THE FOUR STEPS IN THE LOGIC OF 
EVALUATION? 



Scriven (1969, 1989) published a variety of writings on the topic of the logical sequence 
of concepts that defines how people try to connect data to value judgments that the 
evaluand is good or bad, better or worse, passing or failing, or the like. Scriven outlined 
the four steps in 1980: 

1 . selecting criteria of merit, those things the evaluand must do to be judged good 

2. setting standards of performance on those criteria, comparative or absolute levels 
that must be exceeded to warrant the appelation "good" 

3. gathering data pertaining to the evaluand's performance on the criteria relative to the 
standards 

4. integrating the results into a final value judgment. To the extent that evaluation really 
is about determining value, some version of this logic ought to be universally applicable 
to the practice of evaluation. 

ARE QUALITATIVE EVALUATIONS VALID? 
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More qualitative theorists than not seem to both use the word, subject to validity 
criticism, and endorse some version of its applicability. However, some qualitative 
theorists reject both the term and any cognates who seem to garner attention 
disproportionate to their representation in their own field. From outside the qualitative 
camps, the answer also seems to be more uniformly "yes." Nevertheless, the subtleties 
required for an intelligent discussion of this question are extensive, of which the 
following few will illustrate but not exhaust. Even those who reject the concept of 
"validity" will acknowledge they are concerned in their work to "go to considerable pains 
not to get it all wrong" (Wolcott, 1990, p. 127). Further, within and across those methods 
qualitative theorists often disagree among themselves. In addition, qualitative methods 
often aim to produce knowledge of a substantively different kind than other methods, so 
that particular validity criteria may be less pertinent to the interests of qualitative 
evaluations. Indeed, it would be wrong to assume all qualitative methods are alike. 
Different qualitative methods may have different aims that bring different validity criteria 
to bear. In the end, though, some version of validity as an effort to "go to considerable 
pains not to get it all wrong"(Wolcott, 1990, p. 127) probably underlies all methods used 
by all evaluators, quantitative and qualitative. 

WHAT DIFFERENCE DOES IT MAKE WHETHER 
THE PROGRAM BEING 

EVALUATED IS NEW OR HAS EXISTED FOR MANY YEARS?Rossi and Freeman 
(e.g., 1993) long made this distinction central to their approach to evaluation because it 
has several implications for evaluation practice. Brand-new programs have not yet had 
time to work out program conceptualization and implementation problems. Thus, a 
focus on those kinds of questions is likely to be more useful and more acceptable to 
program staff than a focus on, say, outcome questions. In addition, less background 
information and fewer past evaluations are likely to exist for new programs, so more 
work will have to be done "from scratch." Well-established programs may be more ready 
for outcome evaluation, and they may have a greater wealth of information already 
available on them. However, long-established programs may also have reached so 
many of the potential participants that outcome evaluations might be thwarted by 
difficulty finding appropriate control group participants if a controlled design is used. 

IS THERE A DIFFERENCE BETWEEN 
EVALUATING A LARGE PROGRAM, A 

PROJECT WITHIN THAT PROGRAM, OR A SMALL ELEMENT WITHIN 

THAT PROJECT?This distinction points to an interesting tradeoff between ease and 
frequency of short-term change on the one hand, and likely impact on the other (Cook, 
Leviton, & Shadish, 1985; Shadish, Cook, & Leviton, 1991). Small elements have 
natural turnover rates that are much more frequent than for local projects, which 
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themselves turnover less often than large programs. Hence, the opportunity to change 
each of them by replacement is more frequent for smaller than larger entities. However, 
smaller entities are usually likely to have a smaller impact on the overall set of problems 
to which the program, project, or elements are aimed. All this has implications for the 
kinds of questions worth asking depending on what kind of use and impact is desired. 

HOW CAN THE CHANCES OF EVALUATION 
RESULTS BEING USED IN THE 

SHORT-TERM TO MAKE CHANGES BE INCREASED?The literature on this topic is 
extensive (e.g., Cousins & Shuhla, 1997; Patton, 1997), and includes advice to locate a 
powerful user(s), identify questions of interest to the user(s), focus on things that the 
user has sufficient control over to change, discuss exactly what changes the user(s) 
would make given different kinds of answers that might result from the evaluation, 
provide interim findings at points when they might be useful, consider reporting results 
in both traditional and nontraditional formats, provide brief executive summaries of 
results, have continued personal contact after the evaluation ends, and lend support to 
subsequent efforts to foster use of evaluation results. 

WHAT ARE THE DISADVANTAGES OF 
FOCUSING ON SHORT-TERM 



INSTRUMENTAL USE?There is a risk that the evaluation will focus on less important 
interventions or questions than might otherwise be the case, and lose the big picture or 
the long-term outlook about what is important. In part this reflects the tradeoffs 
discussed regarding the program-project-element distinction because instrumental use 
is more likely with smaller elements to have less impact. It also reflects the fact that the 
modern industrial societies where much evaluation takes place have often solved the 
easiest problems, so those that remain are often difficult to do anything about in the 
short-term. Those things that can be addressed in the short-term are rarely likely to fall 
into the set of most difficult problems. Finally, it is rare to find a user who can control 
options that promise truly powerful or fundamental changes. 

WHAT ROLE DOES CAUSAL INFERENCE PLAY 
IN EVALUATION? 



The most obvious version of this question concerns the role of outcome evaluation. 

From an early dependency on outcome evaluation as paradigmatic for the field, the field 
realized the value of asking a wide array of other questions depending on contingencies 
like those discussed previously regarding use, program size, and stage of program 
development So causal inference of the traditional sort assumed a smaller role in 
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evaluation than in early years. Another version of this question appeals to the distinction 
between descriptive causal inferences and causal mediation; the latter has enjoyed 
some recent resurgence in some kinds of theory-driven evaluation. 

WOULD THE ANSWER CHANGE IF QUESTIONS 
WERE ASKED ABOUT THE 

ROLE THAT CAUSAL INFERENCE PLAYED IN MAKING VALUE JUDGMENTS?Most 
readers probably assumed "evaluation" in the previous question to mean the wide range 
of activities that fall under the rubric of professional evaluation practice. This question 
plays on limiting the meaning of the term "evaluation" to the activity of making a value 
judgment. Some readers might not realize that, even in this limited context, causal 
inference still plays an important role. Referring back to the answer to the first question 
about the logic of evaluation, implicit in it in most applications is that the thing being 
evaluated caused the observed performance on the criteria of merit (e.g., that the 
treatment met recipient needs). If that were not the case, it would be improper to 
attribute the merit or value to the evaluand; rather, it should be attributed to whatever 
else actually caused the improvement in the criteria of merit. Thus attributing merit or 
worth is frequently causal in substantial part. 

WHEN DOES A QUESTION HAVE LEVERAGE? 

Cronbach and his colleagues (Cronbach, Ambron, Dornbusch, Hess, Hornik, Phillips, 
Walker, & Weiner, 1980) used this term to describe questions they thought particularly 
worth asking because of their potential for high payoff. Such questions have little prior 
information available, they can be feasibly answered with the resources and in the time 
available, the answers will probably reduce uncertainty significantly, and the answers 
are of interest to the policy shaping community. 

WHAT IS METAEVALUATION, AND WHEN 
SHOULD IT BE USED? 



Metaevaluation is the evaluation of evaluation (Cook & Gruder, 1978; Scriven, 1969), 
and recommendations vary from doing it for every evaluation to doing it periodically. 

The general prescription is that metaevaluation can be done using the same general 
logic (and sometimes methods) for doing the primary evaluation. One might apply the 
logic of evaluation from the first question, for example, asking what the evaluation would 
do well to be a good evaluation (e.g., would it be useful, true, important?), deciding how 
well would it do so (e.g., how useful? true by what standards?), measuring the 
performance of the evaluation in these regards, and then synthesizing results to reach a 
judgment about the merits of the evaluation. Metaevaluation can be applied at nearly 
any stage of an evaluation, from evaluating its planned questions and methods, to a 
mid-evaluation review, to evaluating the completed report by submitting it to 
independent consultants and critics. 
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